Model development workflow

Model development workflow

Factors to guide model selections

Example model selection factors¹

• S: Successful convergence. S = Successful, E = Potential error during model run.
• C: Successful covariance step. S = Successful, E = Error during covariance step
• OFV: Objective function value.
• dOFV: Difference of objective function value, relative to the reference model.
• AIC: Akaike’s Information Criteria.
• BIC: Bayesian Information Criteria.

Nesting

Concept: A model is nested within another model if the larger model reduces exactly to the smaller model by setting part of the larger model to its null value, without changing the interpretation of remaining parameters. - Parameters - One- vs. two-compartment model - Add or remove an ETA on a parameter - Covariates - Setting body weight to its reference value, for example, 70 kg. - Setting sex to its reference group, for example, male.

Example of nesting models: one vs two-compartment models

One-compartment model: \[ \frac{dC_p}{dt} \times V1 = -CL \times C_p \]

Two-compartment model: \[ \frac{dC_p}{dt} \times V1 = -CL\times C_p-Q \times (C_p - C_t) \\ \frac{dC_t}{dt} \times V2 = Q \times (C_p - C_t) \]

• If fixing \(Q=0\) (i.e., no distribution), then structurally, a two-compartment model (the larger model) reduces to a one-compartment model (the smaller model).
• One-compartment model is nested within two-compartment model.

Example of nesting models: add or remove an ETA

• Model 1: CL = THETA(1)
• Model 2: CL = THETA(1) * EXP(ETA(1))
• Fixing \(\omega^2_{(1,1)}\) (variance of ETA(1) distribution):
  • No between subject variability (i.e., zero variance)
  • Model 2 (the larger model) reduces to Model 1 (the smaller model).
  • Model 1 is nested within Model 2.

Nesting model comparison: likelihood ratio test (LRT)

• If two models are nested, then we can use likelihood ratio test (LRT) to test if the difference in model fittings is statistically significant.
  • In this case, goodness of model fitting is represented by the objective function value (OFV) of each model.
  • OFV calculation details: lecture in week 10
• Assumptions:
  • The difference in the OFV between two nested models (dOFV) is approximately Chi-square distributed with degree of freedom (DF) equals to the number of additional parameters.
    • One vs two-compartment model: DF=2 (CL and V1 vs. CL, V1, Q, and V2).
    • Add one more ETA: DF=1

Nesting model comparison: LRT test criteria

Nesting model comparison: LRT test criteria - in practice

• Therefore, in practice, instead of using strict Bonferroni corrections, more stringent cutoffs were arbitrarily adapted:
  • A covariate is included if \(dOFV \geq 6.63\) (i.e., \(\alpha = 0.01\) with DF=1).
  • A covariate is included if \(dOFV \geq 10.8\) (i.e., \(\alpha = 0.001\) with DF=1).

library(tidyverse)
data <- data.frame(n = rep(seq(1, 10, 1), 3), 
                   alpha = c(rep(0.05, 10), rep(0.01, 10), rep(0.001, 10))) %>% 
  mutate(p = 1-(1-alpha)^n) %>% 
  mutate(`P value`=case_when(
    alpha==0.05~"P value ≤ 0.05 (dOFV ≥ 3.84)", 
    alpha==0.01~"P value ≤ 0.01 (dOFV ≥ 6.63)", 
    alpha==0.001~"P value ≤ 0.001 (dOFV ≥ 10.8)"))

ggplot(data, aes(x=n, y=p, group=`P value`, color=`P value`))+
  geom_point()+geom_line()+
  xlab("Number of multiple testing")+
  ylab("Probability of ≥1 false positive")+
  theme_bw()

Example of non-nesting models: one-compartment PK vs TMDD models

A one-compartment PK model:

\[ \frac{dC_p}{dt} \times V1=-CL\times C_p \]

A target-mediated drug disposition (TMDD) model:

\[ \frac{dC_p}{dt} = -K_{on} \times C_p \times R + K_{off} \times RC \\ \frac{dR}{dt} = K_{syn}-K_{deg}\times R-K_{on} \times C_p \times R + K_{off} \times RC \\ \frac{dRC}{dt} = K_{on} \times C_p \times R - K_{off} \times RC \] We cannot reduce one model to make it the same as the other via fixing part of the model to null value (i.e., non-nesting models).

Non-nesting model comparison

Two metrics useful to compare non-nesting models: - Akaike Information Criterion (AIC): \(AIC = -2 \times ln(L) + 2 \times K\)¹ - Penalizes likelihood with number of parameters. - Bayesian Information Criterion (BIC): \(BIC = -2 \times ln(L) + K \times ln(N)\)² - Penalizes parameters more heavily as sample size N increases. - Tend to prefer simpler model when N is large.

Non-nesting model comparison

• Both AIC and BIC cannot be used for statistical test (No P values).
• Comparison AIC and BIC need pre-defined criteria.